1. Field of the Invention
The present invention relates to an image processing device and more particularly relates to an image processing device for enlarging or reducing an image by an arbitrary scale by carrying out interpolation of pixel data utilizing a filter coefficient set, of filter coefficient sets corresponding to each phase at the time of dividing the intervals of pixels of an original image by a prescribed dividing number, that is closest the phase of the pixel to be interpolated.
2. Description of Related Art
The use of Cathode Ray Tubes (hereinafter abbreviated to "CRTs") in displays such as televisions etc. is prevalent, with the handling of analog image signals compatible with various image methods and the changing of horizontal scanning frequency to attain compatibility when displaying these images being common.
However, when handling digital image signals, image resolutions often differ depending on the broadcast transmission method as in the case of, for example, NTSC (National Television System Committee) or PAL (Phase Alternation Line-rate), with the number of pixels in the horizontal and vertical directions of images digitized by these methods therefore differing with each method of broadcast transmission. As there are many broadcast methods including HDTV (High Definition TeleVision) etc., there are therefore many pixel number (resolution) standards. Because of this, it is therefore necessary for a system carrying out digital processing on image data to be compatible with all of these transmission methods and it is therefore necessary for the number of pixels to be converted by an "interpolation filter".
Further, as the number of pixels for a displayed image is also fixed at a prescribed number for liquid displays and plasma displays that have also recently become widespread, an interpolation filter for converting the number of pixels of a source image to a number of pixels compatible with these displays is also required.
Next, a description is given of an example of an interpolation filter for converting the number of pixels for an image.
First, a description will be given of the conversion of the enlargement or reducing of the image, and the sampling frequency (number of pixels).
The cases of either enlarging or reducing an image or converting (conversion across image standards of differing resolutions) the sampling frequency (number of pixels) of the image are all realized by carrying out calculations to obtain data for pixels that did not exist in the original image with respect to the position of each pixel of the original image. It is therefore possible to carry out the above two processes by utilizing an interpolation filter for carrying out the same arithmetic operations.
FIG. 1 shows an example of a portion of an original image, with circles in the center of the drawing indicating the position of pixels. This portion includes 8 horizontal pixels and 6 vertical pixels (the number of pixels has been taken to be small for simplicity).
Next, a description is given of the case where a source image is enlarged by the scale of, for example, 10/7. This scale is expressed as a ratio of lengths rather than as a surface area. In the case of the enlargement of the image of FIG. 1, the pixel arrangement (i.e. the pixel spacing etc.) is kept the same as in FIG. 1 so that the displayed image standard does not change. The resulting image after this enlargement is carried out is shown in FIG. 2. As the scale in this case is 1.429 (=10/7) the length of one side of the image is increased by 1.429 times and the number of pixels is increased by approximately 1.429.sup.2.
With respect to, for example, the horizontal direction, there are 8 pixels in the horizontal direction in the original image, with this increasing to 11 or 12 (adjusted to be close to 8.times.10/7=11.429) after enlargement. The positional relationship of each of the pixels corresponding to the same portions of the image in the analogous image after enlargement is therefore different to the positional relationship occurring in the original image. The values for data (expressing luminance and color) for each of the pixels after enlargement are therefore different with those of the original image.
FIG. 3 shows the positional relationship of pixels for the horizontal direction of an original image and an image after enlarging in the case of enlarging an image by a scale of 10/7.
In FIG. 3, Ri (i=1, 2, . . . ) on the upper side shows pixel data for the original image and Qi (i=1, 2, . . . ) on the lower side represents interpolation pixel data after enlargement. A pixel corresponding to Ri is then arranged at a spacing that is 10/7 times that of the spacing for the pixel corresponding to Qi. FIG. 3 only shows the situation for enlargement in the horizontal direction, but the situation is the same for enlargement in the vertical direction, and a description thereof will be omitted.
The values for the data for each of the pixels after enlargement is calculated by interpolation filter operations, i.e. carrying out convolution operations on interpolation coefficients from the values for the pixel data for several peripheral source images in response to the corresponding relationship of the positions of each of the pixels of the source image show n in FIG. 3.
Next, the case of scaling the sampling frequency by, for example, 10/7 without changing the size of the image is described. Changing the sampling frequency in this way is the equivalent of changing the resolution to a higher image standard by a scale of 10/7, i.e. the number of pixels in the horizontal direction is changed by 10/7. In this case, the source image of FIG. 1 is changed one-dimensionally to an image having approximately 1.429 times the number of pixels, i.e. an image having 1.429.sup.2 times the surface density, as shown in FIG. 4.
The corresponding relationship of each of the pixels of FIG. 1 and each of the pixels of FIG. 2 and the corresponding relationship of each of the pixels of FIG. 1 and each of the pixels of FIG. 4 is the same in both cases, as shown in FIG. 3. The arithmetic operation for converting to an image standard where there are more pixels is the same as the arithmetic operation for enlarging an ag image
Next, a description is given of the case of reducing the source image of FIG. 1 by a scale of, for example, 10/13.
As the image standard is not changed in the case of reducing the image, the arrangement of the pixels occurring in the image after reducing, i.e. the pixel spacing etc. is the same as for the source image shown in FIG. 1.
FIG. 5 shows the source image of FIG. 1 reduced by a scale of 10/13. In this case, the scaling factor is 0.769 (=10/13). The length of one side of the image is therefore reduced by 0.769 times and the number of pixels comprising the reduced image is reduced by approximately 0.769.sup.2.
For example, the number of pixels in the horizontal direction of the source image is 8 but after reducing this becomes 6 or 7 (adjusted to be near 8.times.10/13=6.154). The positional relationship of each of the pixels corresponding to the same portions of the image occurring in the analogous image after reducing is therefore different from the positional relationship of each of the pixels occurring in the original image, with the values for the data (expressing luminance and color) for each of the pixels after reducing therefore being different to those of the original image.
FIG. 6 shows the relationship between the pixels for the horizontal direction in the original image and the image after reducing when the image is reduced to a scale of 10/13.
In FIG. 6, Ri (i=1, 2, . . . ) of the upper side represent pixel data of the source image and Qi (i=1, 2, . . . ) of the lower side represent interpolation pixel data for after reducing. The pixels corresponding to Ri are arranged at a pixel spacing that is 10/13 times the spacing of the pixels corresponding to Qi. FIG. 6 shows just the situation for reducing in the horizontal direction and as the situation is the same for the vertical direction, the case for the vertical direction will be omitted.
The values for the data for each of the pixels after reducing is calculated by interpolation filter operations, i.e. carrying out convolution operations on interpolation coefficients from the values for the pixel data for several peripheral source images in response to the corresponding relationship of the positions of each of the pixels of the source image shown in FIG. 6.
Next, the case of scaling the sampling frequency by, for example, 10/13 without changing the size of the image is described. Changing the sampling frequency in this way is the equivalent of changing the resolution to a higher image standard by a scale of 10/13, i.e. the number of pixels in the horizontal direction is changed by 10/13. In this case, the source image of FIG. 1 is changed one-dimensionally to an image having approximately 0.769 times the number of pixels, i.e. an image having 0.769.sup.2 times the surface density, as shown in FIG. 7.
The corresponding relationship of each of the pixels of FIG. 1 and each of the pixels of FIG. 5 and the corresponding relationship of each of the pixels of FIG. 1 and each of the pixels of FIG. 7 is the same in both cases, as shown in FIG. 6. The arithmetic operation for converting to an image standard where the resolution is lower is the same as the arithmetic operation for reducing an image.
When, in the above, an image is enlarged or reduced, or a sampling frequency (number of pixels) is converted, an interpolation filter calculating pixel data for values that do not exist in the original image is necessary.
Next, a description is given of the operations carried out at the interpolation filter.
When, as shown in FIG. 8, the sampling interval of the original image is taken to be S and a position distanced by a distance (phase) P from the position of the pixel R of the source image is taken to be the position (interpolation point) of the pixel Qi generated by interpolation, a value for the pixel Qi is calculated by performing a convolution operation on the value R of the pixel for the peripheral source image.
According to sampling theory, when ideal interpolation is carried out, equation (1) and a sinc function shown in FIG. 9A are taken as an interpolation function f(x), and a convolution operation for pixels from an infinite time in the past to pixels for an infinite time in the future is carried out. EQU f(x)=sin c(.pi..times.x)=sin (.pi..times.x)/(.pi..times.x) (1)
where .pi. is the ratio of the circumference of a circle to its diameter.
However, as it is necessary in reality to calculate the interpolation values within a finite time, an approximate interpolation coefficient is utilized within a finite range.
A nearest, approximation method, bilinear approximation method and cubic approximation method are well known as approximation methods.
In the nearest approximation method, data for one pixel after interpolation is calculated from data for one pixel of source image utilizing the kind of interpolation function shown in equation (2) and FIG. 9B. The variable x of equation (2) and FIG. 9B expresses displacement in the horizontal direction from the pixel position in the original image as a normalized amount using the sampling interval of the original image. ##EQU1##
In the bilinear approximation method, data for one pixel after interpolation is calculated from data for two pixels of the original image utilizing the interpolation function denoted by equation (3) and shown in FIG. 9C. The variable x in equation (3) and FIG. 9C represents displacement in the horizontal direction from the pixel position in the original image as an amount normalized by the sampling interval of the original image. Further, the bilinear approximation method is a well-known method for the linear interpolation method for calculating weighted mean. ##EQU2##
In the cubic approximation method, data for the first pixel after interpolation is calculated from data for four pixels of the original image utilizing the interpolation coefficient shown in equation (4) and FIG. 9D. The variable x of equation (4) and FIG. 9D expresses displacement in the horizontal direction from the pixel position in the original image as a normalized amount using the sampling interval of the original image. ##EQU3##
The convolution operations with the above approximation methods can be carried out by utilizing a so-called FIR (Finite Impulse Response) digital filter. In this operation, sampled values for an interpolation function, which are sampled by the number equal to the prescribed number of the pixels at the sampling points for the original image in the neighborhood of the interpolation point with the interpolation point fitted at the center of the interpolation function, are used as an interpolation filter coefficient set.
For example, when operations are performed using the bilinear approximation method, when the phase P is 0.0, the two weightings (filter coefficients) comprising the filter coefficient set are 1.0 and 0.0, and the coefficient set is therefore the values for the pixel data for the original image with which this position coincides outputted as is.
When the phase P is 0.5, the two filter coefficients are 0.5 and 0.5, and when P is 0.3, the two filter coefficients are 0.7 and 0.3.
In the case of carrying out interpolation operations using the Cubic approximation method, when the phase P is 0.0, the four weightings (filter coefficients) comprising the filter coefficient set are 0.0, 1.0, 0.0 and 0.0 and the coefficient set is the pixel data values for the original image with which this position coincides outputted as is.
Further, when the phase P is 0.5, the four filter coefficients are -0.125, 0.625, 0.625 and -0.125, and are -0.063, 0.847, 0.363 and -0.147 when P is 0.3.
However, as the phases P of pixels of the source image are different for each interpolation point calculated for the data at this time, a plurality of filter coefficient sets corresponding to different phases are required.
Next, a description is given of a related interpolation filter operation device.
FIG. 10 shows an example configuration of an operation device utilizing an interpolation operation, i.e. utilizing an FIR digital filter for carrying out interpolation function convolution operations. The operation device of FIG. 10 carries out convolution operations by utilizing a Cubic approximation method.
A coefficient memory 1 holds a plurality of filter coefficients corresponding to each interpolation point (or each phase) and outputs four filter coefficients FC1, FC2, FC3 and FC4 corresponding to filter select signals supplied by a prescribed device (not shown in the drawings) to multipliers 3-1 to 3-4.
A register 2-1 holds data supplied by a prescribed device (not shown in the drawings) for outputting to a register 2-2 in accordance with a control signal. Registers 2-2 and 2-3 hold data supplied from register 2-1 and 2-2 for outputting to registers 2-3 and 2-4 in accordance with a control signal and register 2-4 holds data supplied by register 2-3.
Registers 2-1 to 2-4 are connected in series so as to operate as a four stage shift register. Horizontally scanned input image data strings are then sequentially inputted in word units to this shift register so that pixel data for four consecutive source images is stored.
In the Cubic approximation method, interpolation data is then calculated from data for a total of four pixels that are two pixels to the left and right that sandwich the interpolation point utilizing a four stage shift register.
The multiplier 3-i (i=1, . . . , 4) multiplies the value stored in the register 2-i and the value (filter coefficient) FCi supplied by the coefficient memory 1 and outputs the result to an adder 4.
The adder 4 calculates the total of the values supplied by the multipliers 3-1 to 3-4 and outputs this total as the interpolation value.
The sum of the products of the data and filter coefficients inputted to the register 2-1 in this way along a time series is calculated at the multipliers 3-1 to 3-4 and the adder 4 with data for interpolation points that is the result of this operation being outputted along a time series.
A description will now be given of the operation of the operation device of FIG. 10 when an original image is enlarged using Cubic approximation by 10/7.
With 10/7 times enlargement of an image, the positional relationship of each of the pixels with respect to each interpolation point in the horizontal direction is set as shown in FIG. 8 and interpolation filter operations are performed.
FIG. 11 shows the values possessed by each part of the operation device of FIG. 10 occurring in each cycle.
With the hardware device for carrying out processing shown in FIG. 10, latency (delays for realizing high-speed operations) usually occurs due to pipeline processing occurring at portions for these multiplication and sum operations, but in this case it is assumed for simplicity that there is no latency.
In the first cycle of FIG. 11, input data R1 that is image data for one pixel portion of the source image is supplied by a prescribed device. At this time, the registers 2-1 to 2-4 hold data Rm0 that is one previous to the input data R1, data Rm1 that is two previous to the input data R1, data Rm2 that is three previous to the input data R1 and data Rm3 that is four previous to the input data R1.
As the value of the control signal at this time is "H", the registers 2-1 to 2-4 shift the respective data on the rising edge of the following clock.
As the data is then respectively shifted, in the second cycle, the registers 2-1 to 2-4 hold R1, Rm0, Rm1 and Rm2. Further, as the value of the control signal at this time is "H", the registers 2-1 to 2-4 then shift data on the rising edge of the following clock.
As the data is then respectively shifted again, in the third cycle, the registers 2-1 to 2-4 hold R2, R1, Rm0 and Rm1. Moreover, as the value of the control signal at this time is "H", the registers 2-1 to 2-4 then shift data on the rising edge of the following clock.
The data is then similarly shifted, so that in the fourth cycle, the registers 2-1 to 2-4 hold R3, R2, R1 and Rm0. A filter select signal P0 showing the phase of the interpolation value Q1 with respect to Rm0, R1, R2 and R3 of FIG. 3 is then supplied to the coefficient memory 1 so that the filter select signal is then supplied so as to correspond to the phase P of the output signal Qi.
In this case, Ra of FIG. 8 corresponds to Rm0, Rb corresponds to R1, Rc corresponds to R2 and Rd corresponds to R3, with Q of FIG. 8 corresponding to interpolation value Q1.
The coefficient memory 1 stores the ten kinds of filter coefficient sets shown in FIG. 12 and selects and outputs four coefficients FC1, FC2, FC3 and FC4 in response to the supplied filter select signal Pi. As there are only the ten types shown in FIG. 3 for the phase of the pixels occurring in this interpolation operation when enlarging the image by a scale of 10/7, the coefficient memory 1 only has a number of filter coefficient sets equal to this number of phases.
Namely, the filter select signal Pi corresponds to the filter coefficient set of the ten types of phases corresponding to each position where S in FIG. 8 is divided into ten equal parts of phase i/10. The decimal point representation coefficient (filter coefficient) of FIG. 12 is a value calculated by taking the phase corresponding to the filter select signal Pi as x and substituting this in equation (4). The eight bit representation coefficient is calculated by limiting this decimal point representation coefficient to an eight bit word length (here the maximum amplitude is taken to be 128).
In this case, the coefficient memory 1 outputs the filter coefficient set (0.0, 1.0, 0.0, 0.0) corresponding to the phase P0 of FIG. 12 (0, 128, 0, 0) when expressed using eight bits) to the multipliers 3-1 to 3-4 as the four filter coefficients FC1, FC2, FC3 and FC4 because the filter select signal is P0.
The product sum operation is then carried out by the multipliers 3-1 to 3-4 and the adder 4, with the results of this operation being outputted as the output data Q1.
At this time (in the fourth cycle), the registers 2-1 to 2-4 do not output the held data on the following clock because the value of the control signal is "L".
In the fifth cycle, the registers 2-1 to 2-4 continue to hold the data held in the fourth cycle. At this time, a filter select signal P7 indicating the phase of the output data Q2 with respect to Rm0, R1, R2 and R3 of FIG. 3 is supplied to the coefficient memory 1.
In this case, as shown in FIG. 3, R1 and Q1 are the same phase and the interval for Q1 and Q2 is 7/10 that of the interval S for R1 and R2. The phase is therefore 7/10 and the filter select signal P7 is supplied.
The coefficient memory 1 then outputs the filter coefficient set (-0.147, 0.363, 0.847, -0.063) (when expressed as eight bits this is (-19, 46, 108, -8)) corresponding to the phase of 7/10 of FIG. 12 to the multipliers 3-1 to 3-4 as four filter coefficients FC1, FC2, FC3 and FC4 because the filter select signal is P7.
The aforementioned product sum operation is then carried out by the multipliers 3-1 to 3-4 and the adder 4, with the result being outputted as output data Q2.
As the pixels for the source image used in computing the output data Q1 for the fourth cycle and the output data Q2 for the fifth cycle are the same, the value for the control signal for the fourth cycle is taken as "L" and the registers 2-1 to 2-4 do not undergo a shift operation at the time of the transition to the fifth cycle.
As, at this time (i.e. in the fifth cycle) the value for the control signal is "H", the registers 2-1 to 2-4 shift data of the rising edge of the following clock.
In the sixth cycle, the registers 2-1 to 2-4 hold R4, R3, R2 and R1, respectively. A filter select signal P4 indicating the phase of the output data Q3 with respect to R4, R3, R2 and F1 of FIG. 3 is then supplied to the coefficient memory 1.
The phase for this occasion becomes 14/10 with 7/10 cumulated to the previous phase of 7/10, but, with subtraction of the phase corresponding to one original image data (=10/10), the phase on this occasion becomes 4/10 (=7/10+7/10 -10/10).
The phase therefore primarily changes by 7/10 at a time each one cycle and as the integral portion is then handled as a data delay, the phase becomes a modulo operation.
The coefficient memory 1 then outputs the filter coefficient set (-0.096, 0.744, 0.496, -0.144) (when expressed as eight bits this is (-12, 95, 63, -18)) corresponding to P4 of FIG. 12 to the multipliers 3-1 to 3-4 as four filter coefficients FC1, FC2, FC3 and FC4.
The above product sum operation is then carried out by the multipliers 3-1 to 3-4 and the adder 4, with the results being outputted as the output data Q3.
At this time, the registers 2-1 to 2-4 shift data on the rising edge of the following clock because the value for the control signal is "H".
The output data Qi is then sequentially outputted in a similar manner as shown in FIG. 11 as the process progresses.
When the number of pixels for an image is changed using the above device, the input data rate and output data rate also changes as a result of changing this number of pixels.
For example, in a conversion with an increase in the number of pixels, although the output data rate is fixed, supply of the input data string is halted in some cases as shown in the fifth cycle of FIG. 11. Further, in a conversion with a decrease in the number of pixels, although the input is fixed, the outputting of output data is halted in some cases.
The data rate can therefore be kept fixed by providing a buffer memory for temporarily storing input/output data for the operation device shown in FIG. 10.
Reducing or enlargement of the image and changing of the resolution is then carried out using hardware (i.e. by using an electronic circuit constructed so as to correspond to each of the operations occurring in the processing).
However, when an image is enlarged or reduced or resolution is changed using a hardware device, it is preferable for processing to be carried out simultaneously during image conversion. This means that in order to carry out various processes such as image processing, television signal processing and noise removal etc. devices corresponding to each process have to be provided separately. As a plurality of devices therefore have to be provided it becomes difficult to make the overall scale of the device small.
As it is the object of the present invention to resolve the aforementioned problems, the present invention sets out to provide an appropriate image processing device that can resolve the aforementioned problems while remaining small.